Effects of Noise on Convergence and Generalization in Recurrent Networks
نویسندگان
چکیده
We introduce and study methods of inserting synaptic noise into dynamically-driven recurrent neural networks and show that applying a controlled amount of noise during training may improve convergence and generalization. In addition, we analyze the effects of each noise parameter (additive vs. multiplicative, cumulative vs. non-cumulative, per time step vs. per string) and predict that best overall performance can be achieved by injecting additive noise at each time step. Extensive simulations on learning the dual parity grammar from temporal strings substantiate these predictions.
منابع مشابه
An analysis of noise in recurrent neural networks: convergence and generalization
Concerns the effect of noise on the performance of feedforward neural nets. We introduce and analyze various methods of injecting synaptic noise into dynamically driven recurrent nets during training. Theoretical results show that applying a controlled amount of noise during training may improve convergence and generalization performance. We analyze the effects of various noise parameters and p...
متن کاملEeects of Noise on Convergence and Generalization in Recurrent Networks
We introduce and study methods of inserting synaptic noise into dynamically-driven recurrent neural networks and show that applying a controlled amount of noise during training may improve convergence and generalization. In addition, we analyze the eeects of each noise parameter (additive vs. multiplicative, cumulative vs. non-cumulative, per time step vs. per string) and predict that best over...
متن کاملAn Analysis of Noise in Recurrent Neural Networks: Convergence and Generalizationy
There has been much interest in applying noise to feedforward neural networks in order to observe their eeect on network performance. We extend these results by introducing and analyzing various methods of injecting synaptic noise into dynamically-driven recurrent networks during training. We present theoretical results which show that applying a controlled amount of noise during training may i...
متن کاملThe role of Membership and Participation in Admissible Virtual Social Networks on Cultural Convergence (Case study: the Young Members of FACE NAMA in Qom)
The appearance of different social media has been very influential in individuals’ life. The most common social networks are virtual social networks which have been fascinated by many users because of a lot of similarities with human society, and communications and interactions which are formed in it. This study investigates the role of membership and participation in virtual social network on ...
متن کاملRegularizing Recurrent Networks - On Injected Noise and Norm-based Methods
Advancements in parallel processing have lead to a surge in multilayer perceptrons’ (MLP) applications and deep learning in the past decades. Recurrent Neural Networks (RNNs) give additional representational power to feedforward MLPs by providing a way to treat sequential data. However, RNNs are hard to train using conventional error backpropagation methods because of the difficulty in relating...
متن کامل